HEVC (High Efficiency Video Coding) is an advanced video coding system being developed under the Joint Collaborative Team on Video Coding (JCT-VC) group of video coding experts from ITU-T Study Group. In HEVC, a 2N×2N coding unit can be hierarchically partitioned into a partition type selected from 2N×2N, 2N×N, N×2N and N×N. The coding system uses a criterion to determine the best partition, where RD-rate is often used as the criterion. The N×N partition at level k is evaluated and the same partition, i.e., 2N×2N partition is also evaluated at level k+1. Therefore, N×N partition at level k becomes redundant if 2N×2N partition at level k+1 will be evaluated. In order to eliminate the above redundancy, the allowable partition sizes are constrained according to the method disclosed in U.S. Non-Provisional patent application, Ser. No. 13/012,811, filed Jan. 25, 2011, entitled “Apparatus and Method of Constrained Partition Size for High Efficiency Video Coding”. In U.S. Non-Provisional patent application, Ser. No. 13/012,811, for each leaf CU larger than the SCU (smallest CU), the partition sizes allowed are 2N×2N, 2N×N and N×2N. In other words, the N×N partition is not allowed for INTER mode if the leaf CU is larger than the SCU. If the leaf CU size is the same as SCU size, all partition sizes, 2N×2N, 2N×N, N×2N, and N×N, are allowed. While the method disclosed in U.S. Non-Provisional patent application, Ser. No. 13/012,811 reduces computational complexity at the expense of modest performance loss, it is desirable to develop a method and apparatus that can further reduce the computational complexity with about the same performance. Furthermore, it is desirable to provide flexibility so that either the method and apparatus with further complexity reduction can be selected or an alternative method and apparatus can be selected.